Automatically Scoring Freshman Writing: A Preliminary Investigation
نویسندگان
چکیده
In this work, we explore applications of automatic essay scoring (AES) to a corpus of essays written by college freshmen and discuss the challenges we faced. While most AES systems evaluate highly constrained writing, we developed a system that handles open-ended, long-form writing. We present a novel corpus for this task, containing more than 3,000 essays and drafts written for a freshman writing course. We describe statistical analysis of the corpus and identify problems with automatically scoring this type of data. Finally, we demonstrate how to overcome grader bias by using a multi-task setup, and predict scores as well as human graders on a different dataset. Finally, we discuss how AES can help teachers assign more uniform grades.
منابع مشابه
Automatically Extracting Topical Components for a Response-to-Text Writing Assessment
We investigate automatically extracting multiword topical components to replace information currently provided by experts that is used to score the Evidence dimension of a writing in response to text assessment. Our goal is to reduce the amount of expert effort and improve the scalability of an automatic scoring system. Experimental results show that scoring performance using automatically extr...
متن کاملOntology-Based Argument Mining and Automatic Essay Scoring
Essays are frequently used as a medium for teaching and evaluating argumentation skills. Recently, there has been interest in diagrammatic outlining as a replacement to the written outline that often precedes essay writing. This paper presents a preliminary approach for automatically identifying diagram ontology elements in essays, and demonstrates its positive correlation with expert scores of...
متن کاملEffect of Holistic vs. Analytic Assessment on Improving Iranian Intermediate EFL Learners’ Writing Skill
In assessing foreign language writing, holistic and analytic scoring can be used to measure a variety of discourse and linguistic features. This study aimed to investigate the possible significant effect of analyt- ic and holistic assessments on improving writing skill among Iranian EFL learners. For this purpose, two groups of intermediate EFL learners, after being homogenized, were divided in...
متن کاملWord Embedding for Response-To-Text Assessment of Evidence
Manually grading the Response to Text Assessment (RTA) is labor intensive. Therefore, an automatic method is being developed for scoring analytical writing when the RTA is administered in large numbers of classrooms. Our long-term goal is to also use this scoring method to provide formative feedback to students and teachers about students’ writing quality. As a first step towards this goal, int...
متن کاملAutomated Scoring of a Summary-Writing Task Designed to Measure Reading Comprehension
We introduce a cognitive framework for measuring reading comprehension that includes the use of novel summary writing tasks. We derive NLP features from the holistic rubric used to score the summaries written by students for such tasks and use them to design a preliminary, automated scoring system. Our results show that the automated approach performs well on summaries written by students for t...
متن کامل